How would you design observability — metrics, logs, and traces — for a Go microservice in production?
Expose Prometheus metrics on /metrics, emit structured JSON logs with slog enriched with trace IDs, and instrument with OpenTelemetry for distributed traces. Correlate all three with a shared trace ID.
Observability setup
Observability pillars
Metrics: request rate, error rate, latency (P50/P95/P99), goroutine count, GC pause time
Logs: structured JSON with traceID, spanID, requestID, userID on every entry
Traces: OpenTelemetry spans for HTTP requests, DB queries, outbound calls — visualized in Jaeger/Tempo
Correlate by injecting traceID into log context: span.SpanContext().TraceID().String()
Stack: Prometheus + Grafana (metrics), Loki (logs), Tempo/Jaeger (traces) — or Datadog/New Relic for managed